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Policy Brief on Data Analyses and Their Implications for School Policy 

Steven E. Stemler, Robert J. Sternberg , 

Elena L. Grigorenko, Linda Jarvin, and Donna Macomber 

One of the most important initiatives in education today is the No Child Left Behind Act (U.S. 
Department of Education, 2003b). Implied within this Department of Education’s legislative mandate is 
the notion that all children should receive the support that they require in order to meet their educational 
goals. Schools are expected to provide this support. Unfortunately, in the United States today, schools are 
not all equally successful at accomplishing their educational mission. In fact, a body of research evidence 
shows that schools often differ substantially in their levels of effectiveness (Arnold & Sedlacek, 1995; 
Mullis, Jenkins, & Johnson, 1994; Teddlie & Reynolds, 2000). For example, using data from the Third 
International Mathematics and Science Study, Stemler (2001) pointed out that nearly one quarter of the 
variability in student achievement in the United States at the elementary school level can be explained 
simply by knowing which school the student attends. If the dominant educational policy goal of this era is 
to ensure that no child is left behind, then it is incumbent upon policymakers to provide students with 
sufficient support structures to help them reach their goals. In order to do this, policymakers must have an 
understanding of three areas. First, it is important to understand the factors that are associated with school 
effectiveness. Second, it is important to understand which of these factors are sensitive to changes in 
educational policy. Third, it is important to understand the types of schools in which changes (whether 
surface-structural or deep-structural changes) in educational policy are likely to be successful. This policy 
brief presents preliminary findings from a project designed to explore these three issues. 

In order to capture more fully the complex workings of school systems, the current project 
employs a mixed-methods approach to data collection. The first set of data was gathered via structured 
interviews with principals and teachers. The interviews were designed to assess implicit theories about the 
important factors that come together to create successful school systems. The second set of data is 
currently being collected through an online survey designed to test the explanatory power of two explicit 
theories of school effectiveness discussed in the next section. 
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BACKGROUND 



Formal research on school effectiveness dates back to the 1960s and is often linked to landmark 
studies by Coleman et al. (1966), Jencks et al. (1972), and the Plowden Committee in England (1967). 
The findings in each of these studies suggested that, after taking into account differences in students’ 
home background, school-related factors did not have much influence on student achievement. These 
results were widely interpreted to mean that schools had little impact on the lives of children. In reaction 
to these findings, educational researchers began devoting a substantial amount of time and resources to 
systematic investigations of school impact. The concept of school effectiveness has been suggested as one 
of the indicators of school impact. 

Reynolds, Teddlie, Creemers, Scheerens, & Townsend (2000) have noted that early school 
effectiveness research was focused on educational inputs, such as school resource variables (e.g., per 
pupil expenditures, school size) and student background variables (e.g., student socioeconomic status). 
The findings from these early studies typically indicated that differences in school effectiveness were 
more closely associated with differences in student background than with school-related variables. Later, 
however, the search for process indicators began to take precedence in school effectiveness research as it 
was found that process indicators accounted for additional variability in student achievement. For 
example, there is some evidence to suggest that instructional variables, such as the amount and frequency 
of homework assigned (Fuller, 1987), the use of technology in the classroom (Mayer, Mullems, & Moore, 
2000; Wenglinsky, 1998), the frequency of group work and/or teacher lectures (Arnold & Sedlacek, 
1995), and the nature of school leadership (Leithwood, 2000), are all associated with student 
achievement. 

In her review of the literature related to school effectiveness, Anderson (1982) classified the 
variables associated with school effectiveness into four broad areas, a classification that continues to 
provide a useful framework for thinking about research in the field even 20 years later. The first area 
relates to physical resources of the school. This area includes variables such as class size, school size, and 
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per-pupil expenditure. A second area relates to milieu: the social dimension of schooling that is concerned 
with the presence or absence of particular groups within the school and includes variables such as parental 
involvement and percentage of minority students in the student body. A third area relates to social 
systems and is concerned with the patterns of interactions among persons and groups within the school. 
Variables related to social systems include the frequency and nature of interactions in the school among 
administrators and teachers, parents and teachers, and teachers and students. Finally, a fourth area relates 
to school culture. This area includes variables related to the relevance and visibility of the school mission 
and the educational philosophy of the teachers. 

Theoretical Framework of the Current Project 

The current project investigates each of the four clusters of school effectiveness factors discussed 
above and introduces a slightly different theoretical framework. Specifically, the current project is based 
on the idea that the multiple dimensions of schooling (e.g., physical resources, social structures, and 
attitudes and beliefs) interact to create a coherent system and a unique identity for each school, much as 
human beings are multifaceted creatures whose various intrapsychic dimensions interact to create an 
overarching identity. As such, we believe that it is possible to test the “intelligence” of this school system. 
For such an understanding of school effectiveness, Sternberg’s (1985, 1997) theory of successful 
intelligence can provide the guiding theoretical framework. According to the theory, intelligence 
comprises three abilities: analytic, creative, and practical. These abilities can be found across all of the 
various subdimensions that interact to create a system. In other words, these abilities are not domain 
specific. Analytic abilities are associated with activities that require systematic reflection related to 
different ways to approach a situation. Creative abilities are associated with activities such as generating 
new ideas, flexible thinking, and synthesizing alternative perspectives on various issues. Practical abilities 
are associated with activities that are driven by functionality, such as using resources adequately and 
applying lessons learned from previous experiences. One goal of the current project is to examine the 
influence upon school effectiveness associated with each of the domains’ analytic, creative, and practical 
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dimensions identified by Anderson’s (1982) framework (with the modification of collapsing the social 
systems category and the milieu category into one overall category called social structure). 

For example, physical resources are typically related to areas such as the cosmetic appearance of 
the school, availability of instructional resources, and the quality of instructional resources. Using the 
theory of successful intelligence, we could imagine indicators that would represent the analytic (e.g., the 
school building is clean and orderly), creative (e.g., teachers can decorate their classrooms according to 
their interests), and practical (e.g., the resources of the school are well used) subcomponents of the 
physical resources dimension. 

Social structures are typically related to areas such as the principal’s leadership style, the school’s 
general approach to educational change, the relationship between the teachers and principal, and the level 
of parental involvement in the school. Here too, we can imagine indicators that would represent the 
analytic (e.g., teachers are given time to learn from each other in a formal way), creative (e.g., the climate 
makes it possible to question procedures or norms and to discuss these with others), and practical (e.g., 
when agreements are made between teachers and administrators, follow-through is poor) subcomponents 
of the social structure dimension. 

Finally, attitudes and beliefs relate to areas such as the role of the teacher, the teacher’s general 
attitude toward the school, and the school’s openness to outsiders. We can imagine indicators that would 
represent the analytic (e.g., people accurately recognize the strengths and weaknesses of the school), 
creative (e.g., teachers encourage students to understand and tolerate different perspectives), and practical 
(e.g., there is a climate of trust in the school among students, teachers, and administrators) subcomponents 
of the social structure dimension. 

The current project also aims to examine the extent to which Sternberg’s (2000, 2002) 
mineralogical theory of school modifiability provides a useful and empirically valid framework for 
thinking about school effectiveness. The mineralogical theory suggests that a key part of school culture 
that ultimately impacts upon schools’ effectiveness relates to each school system’s openness to both 
surface-structural and deep-structural changes. The mineralogical theory requires schools to ask three 
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questions of themselves in order to assess their openness to change, or modifiability. These three 
questions are: 

1 . How much desire is there for actual change within the school culture? 

2. How much desire is there for the appearance of change in the school culture? 

3. What is the perceived quality of the school? 

For the purposes of simplicity, we can think of people responding to each question in a 
dichotomous manner (either high or low), which allows eight different school culture profiles with respect 
to modifiability. Although the eight categories that follow cannot capture the entire range of all possible 
school cultures, we believe that the classifications do capture an important element of school culture 
related to modifiability, which is ultimately linked closely to school effectiveness. 

Modifiability can be of two basic kinds: surface-structural or deep- structural (of course, there is 
an underlying continuum here that has been dichotomized for convenience.). Surface-structural 
modifiability refers to the extent to which an intervention seeks merely to build on what is already present 
in the school. The changes that occur happen in the context of the school as it already exists. Deep- 
structural modifiability refers to the extent to which more profound changes can be achieved. A deep- 
structural intervention requires building new organizational structures, and making at least some 
fundamental changes in the nature of the organization. The modifiability of an organization depends 
primarily upon its desire for actual change, and secondarily on its desire for the appearance of change, 
and its perceived quality. 

Table 1 summarizes the mineralogical theory of school modifiability. The basic characteristics of 
each school profile will be presented briefly here (for a more detailed discussion, see Sternberg, 2000, 
2002 ). 
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Table 1 : Eight Types of Schools in the Mineralogical Theory of School Modifiability 



Description 


Desire for 
Actual 
Change 


Desire for 
Appearance 
of Change 


Perceived 

Quality 


Modifiability 










Surface 


Deep 


Rusted Iron 


L 


L 


L 


L 


L 


Granite 


L 


L 


H 


L 


L 


Amber (with Internal Insects) 


L 


H 


L 


ML 


L 


Opal 


L 


H 


H 


ML 


L 


Cubic Zirconium 


H 


L 


L 


ML 


L 


Slightly Imperfect Diamond 


H 


L 


H 


MH 


ML 


Lead 


H 


H 


L 


MH 


ML 


Diamond in the Rough 


H 


H 


H 


H 


H 



Note: H=High, MH=Moderately High, ML=Moderately Low, and L=Low 



First, the “rusted iron” school ranks low in its desire for actual change, desire for appearance of 
change, and perceived quality of the school. The general attitude is one of hopelessness; the staff simply 
tries to get through each day. Some of the characteristics associated with this type of school include 
entrenched bureaucracy, apathy, decayed physical plant, staff burnout, and poor follow-through on 
agreements. The prospect of both surface-structural and deep-structural change is low. 

Second, the “granite” school ranks low in its desire for actual change and its desire for 
appearance of change, but high in its perceived school quality. The general attitude here is one of 
smugness. The staff believes that the school works, even if it is not well-resourced. Some of the 
characteristics associated with this type of school include an emphasis on discipline, the use of a 
traditional curriculum, pride in unchanging methods, old materials, and grimness in the attitudes and 
beliefs of the personnel. The prospect of both surface-structural and deep-structural change here is also 
low. 

Third, the “amber” school (with internal insects) ranks low in its desire for actual change, high in 
its desire for the appearance of change, but low in its perceived quality. Its mood is one of frustration. The 
staff believes that the core is rotten, but to change it would destroy the entire system, and the staff along 
with it. Some of the characteristics associated with this type of school include an aging administration, 
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hyperstability in the face of dissension, and inaccessibility of the power structure. The prognosis for 
surface change is medium low, and the prognosis for deep-structural change is low. 

Fourth, the “opal” school ranks low in its desire for actual change, but high in its desire for 
appearance of change and its perceived quality. Its mood is one of self-righteousness. The staff believes 
that outward changes are indicative of high quality. Some of the characteristics associated with this type 
of school include affluence, a shiny physical plant, clear emphasis on appearances, high salaries, and a 
surprising lack of mission. The prognosis for surface structural change is moderately low, and the 
prognosis for deep-structural change is low. 

Fifth, the “cubic zirconium” school ranks high in its desire for actual change, but low in both its 
desire for the appearance of change and its perceived quality of the school. The mood of the school is 
fraudulence. The staff aims to keep outsiders at a distance and not let them know what is really happening 
in the school. Some of the characteristics associated with this type of school include resistance to scrutiny, 
history of no research in the school, and an emphasis on show rather than substance. The prognosis for 
surface-structural change is moderately low, and the prognosis for deep-structural change is low. 

Sixth, the “slightly imperfect diamond” school ranks high in its desire for actual change, low in 
its desire for the appearance of change, and high in its perceived quality. The general attitude here is one 
of denial. The staff believes that the school would be in excellent standing if only one thing could be 
changed. The fact, however, is that the “one thing” is used as a scapegoat for all the schooFs woes. Some 
of the characteristics associated with this type of school include praise of the system coupled with veiled 
digs at X (their scapegoat), attempts to deny the problem of X, and generally favorable signs, but subtle 
hints that something is wrong. The prognosis for surface-structural change is moderately high, and for 
deep-structural change is moderately low. 

Seventh, the “lead” school ranks high in its desire for actual change and its desire for appearance 
of change, but low in its perceived quality. The mood here is one of superstition. The staff believes that 
there is a quick fix that will solve the school’s problems. Some of the characteristics associated with this 
type of school include impatience, lack of understanding about interventions, and an emphasis on doing 
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rather than planning. The prognosis for surface-structural change is moderately high, and that for deep- 
structural change is moderately low. 

Finally, the “diamond in the rough” school ranks high in its desire for actual change, desire for 
appearance of change, and perceived quality. The general attitude here is one of hopefulness. The staff is 
open to help from the outside and is always looking for ways to improve. Some of the characteristics 
associated with this type of school include willingness to devote time and money to change, strategic 
planning, and an accurate recognition of the strengths and weaknesses of the school. The prognosis for 
surface-structural and deep-structural change is high. 

Although Anderson’s (1982) review of the literature related to key factors associated with school 
effectiveness remains an important reference in the field, the paper is 20 years old. Consequently, a 
potentially important step in the current investigation was to explore teachers’ and principals’ implicit 
theories of the most basic elements that make schools successful. The interview protocols were designed 
to elicit a rich description of the underlying mechanisms behind each of the four key dimensions, and to 
investigate new dimensions that may have emerged. The structured interviews were intended to answer 
the first two policy questions related to the identification of relevant factors associated with school 
effectiveness, and which of the factors are open to policy manipulation. 

The online surveys are designed to provide a quantitative indicator of the most important factors 
associated with aggregate student achievement. Subscale scores can be computed for a physical resources 
factor, a social structures factor, and an attitudes and beliefs factor. Further, subscale scores can also be 
generated for the creative, analytic, and practical components of each of those factors. These nine 
subscale scores allow for an examination of the usefulness of the intelligent systems framework in the 
prediction of school effectiveness. Finally, the survey data also allowed us to examine the association 
between student achievement and subscale scores associated with each of the eight school profiles 
associated with the mineralogical theory of school modifiability. 
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METHODS 



Interviews 

The 462 public and private elementary, middle, and high schools that were recognized by the 
United States Department of Education as National Schools of Excellence (i.e., Blue Ribbon Schools) 
during the 1999-2000 and 2000-2001 program cycles were used to gather insights for the purposes of 
this project. The 404 public and 58 private schools selected represented 42 states and the Department of 
Defense Education Agency (U.S. Department of Education, 2003a). 

A random sample of 50 Blue Ribbon Schools was selected, and the e-mail addresses of the 
principals from these schools were obtained. E-mail invitations were then sent to the principals and their 
staff to participate in the structured interviews. A total of eight teachers and eight principals were 
interviewed for this segment of the project between March 2002 and June 2002. Each interview followed 
a structured protocol of questions that related to the following three topics: 

1 . Systems Thinking. The questions here included, but were not limited to, the key players that 
make up the system, the key elements that must be present for a school to be successful, and a 
description of stages that schools pass through as they become effective. 

2. Leadership. The questions here included, but were not limited to, how policies are made and 
changed, the role of consensus building, and how various kinds of school problems are 
approached. 

3. School Culture. The questions here included, but were not limited to, the level of cooperation 
among teachers, the general attitude of the staff, the attitudes toward internal versus external 
research, and the amount of time teachers have to reflect on practice. 

The interviews lasted approximately 1 hour each, and all participants were compensated with 
either a $50 gift certificate to Borders Bookstore or a $50 donation to the charity of their choice. 
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Surveys 



In order to test the explanatory power of the two explicit theories of school effectiveness 
described previously (Anderson, 1982; Sternberg, 1997, 2000, 2002), we created the School 
Characteristics Inventory (SCI). This instrument consists of 85 items that may be combined to yield 
separate subscale scores associated with the analytic, creative, and practical components of the physical 
resources, social structures, and attitudes and beliefs of each school. It should be noted that the SCI is a 
work in progress rather than a finished product, and the results of the pilot study will help to refine both 
the theory and its measurement. Table 2 presents the number of items on the SCI designed to represent 
each dimension within the intelligent systems framework. 



Table 2: Total Number of Items on the SCI Assessing Each of the Intelligent Systems Dimensions 





Analytic 


Creative 


Practical 


General 


TOTAL 


Physical Resources 


3 


4 


2 


3 


12 


Social Structures 


6 


10 


11 


4 


31 


Attitudes and Beliefs 


14 


8 


8 


9 


39 


TOTAL 


23 


22 


21 


16 


82 



In addition, the items on the SCI allow for the calculation of subscale scores associated with each 
school profile to relate to the mineralogical theory. Table 3 presents the breakdown of the number of 
items designed to generate each unique school profile. 



Table 3 : Total Number of Items on the SCI Assessing Each Mineralogical School Type 



School Type 


Number of Items 


Rusted Iron 


6 


Granite 


6 


Amber 


8 


Opal 


7 


Cubic Zirconium 


7 


Slightly Imperfect Diamond 


5 


Lead 


5 


Diamond in the Rough 


6 


TOTAL 


50 
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The types of items included in the subscale for each school correspond closely to the 
characteristics of each profile outlined in the theoretical background section of this paper. 

In order to test the predictive power of the questionnaire for discriminating between more and 
less effective schools, we attempted to collect data from a broad range of school types, including urban, 
suburban, and rural schools. As a recruiting strategy for the pilot study, we compiled a list of all of the 
superintendents in each state drawn from various publicly accessible databases on the Internet (e.g., 
National Center for Education Statistics School Locator). Introductory e-mails describing the project were 
sent out to superintendents whose e-mail addresses were available. If a superintendent expressed interest 
in the project, the project director would speak with the superintendent over the phone and e-mail further 
information about the project. The superintendent was next asked to recommend three to five schools in 
their district that might be interested in participating in the project. The principals from these schools 
were then contacted via e-mail by the project director and invited to participate, and asked to distribute e- 
mail to their staff inviting all who were interested to participate. 

Participants were sent a letter explaining the project and including a link to the website where 
they could login to fill out the online survey. The SCI took approximately 30-45 minutes to complete, 
and all participants were compensated with either a $40 gift certificate to Barnes & Noble Bookstore or a 
$40 donation to the charity of their choice. 1 

PRELIMINARY FINDINGS 
Interview Results 

Although the structured interviews yielded a substantial amount of data, for the purposes of this 
policy brief we will discuss only the results related to the three policy questions of interest: (1) key factors 
related to school effectiveness, (2) factors that are open to policy manipulation, and (3) characteristics of 
schools that are open to surface-structural and deep-structural change. 



1 Participants were also asked to complete a Tacit Knowledge Inventory related to their profession as part of this 
segment of the project. 



Key Factors Related to School Effectiveness 



Interviews began with the participants being asked to think about their school as an integrated 
system. They were asked to consider what were the most basic elements necessary to make the system 
successful. The results were quite consistent across both the principals and the teachers. Both sets of 
participants tended to identify elements related to individual personality types as well as interpersonal 
interactions. Although some participants mentioned the need for adequate resources (e.g., papers, pencils, 
access to photocopy machines), these were typically secondary to the interpersonal dimensions. As one 
middle-school principal explained: 

People are the foundation and the sparkle that make everything else go. Without the 
appropriate interactions between students and teachers, teachers and principals, students 
and principals, and all of that, without those dynamics, the curricular issues and such 
suffer. 

Key concepts related to interpersonal interaction had to do with clear communication and mutual 
respect among the teachers and administrators. The notion of respect was defined as open-mindedness, 
flexibility, a willingness to listen to multiple points of view, and a willingness to compromise. Yet, as one 
teacher pointed out, these characteristics do not necessarily imply warmth. 

Although students, teachers, and administrators were often cited as the key players, many 
participants noted the strong influence of parental involvement on the success of the school system. As 
one high-school principal noted: 

I've found that parents usually push depending on the school itself... the school will be as 
good or as progressive or as important to them as they want it to be. So if they're not all 
that interested, schools can tend to be on cruise control and just not really do much more 
than simply exist. Whereas if a parent group, even a small but vocal group, continues to 
push for their agenda or a more global agenda, you can get the school to move in a 
certain direction. 
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Clearly, the parents can also be a key factor associated with school effectiveness. In general, the 
interview results suggest that people with strong interpersonal skills inhabit more effective schools. 
Principals and teachers must be skilled at interacting with several different audiences, including other 
teachers, administrators, parents, and students. 

Factors that are Open to Policy Manipulation 

Many of the participating Blue Ribbon Schools were very well resourced, which may be one 
reason that the physical resources dimension was not often noted as a critical component to school 
success. Nevertheless, one of the interviewed teachers did mention the importance of sufficient resources: 
In getting “materials such as paper,” if there’s not “a real hassle to do it, then your energy 
can be devoted to thinking up a more creative lesson. But if you have to spend all your 
time working here and awaiting this and not having enough of that, and waiting for that 
to come, it wears you down.” 

This quote suggests that once a modicum of resource needs has been met, the teachers have the 
time and energy to think more creatively. Furthermore, a lack of adequate resources may negatively 
impact staff attitudes and beliefs and possibly affect the quality of social interactions in the school. 

Another common theme that was mentioned across interviews was the importance of shared 
commitment to a common vision or philosophy. In fact, the notion of commitment to a shared philosophy 
was tremendously important at one of the Blue Ribbon middle schools in our sample. The response from 
one principal interviewed is suggestive of how this area may lend itself to policy manipulation: 

When we reorganized the middle school, we had a staff that came on that was committed 
to the idea of [a common] philosophy, and we hired people to reinforce that philosophy. 

So I think we had a vision, and then we all came together because we bought it. So I 
think that made us so successful — willing to work, willing to keep up the pace, willing to 
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get the job done. Recognition of each other and our strengths helped us bring the big 
picture. That’s what made it successful. And appreciation of each other. We recognize 
what we do for the kids and then we get each other’s support for it. 

As a nation, “The worst teacher shortage ever is now cooking. Nationwide, some 2.4 million 
teachers will be needed in the next 1 1 years because of teacher attrition and retirement and increased 
student enrollment” (National Education Association, 2002). This means that new teachers need to be 
trained to fill the gaps. The interview results presented here suggest that, as a policy matter, schools 
would be well-advised to explicitly state their educational philosophy, and to attempt to hire teachers who 
will reinforce the shared philosophy of the school. 

Characteristics of Schools that Embrace Surface-Structural and Deep-Structural Change 

When discussing changes that occurred within the school, many teachers noted that changes 
tended to be driven by the school leadership. This is particularly interesting in light of the fact that most 
of the teachers we spoke to had been at their respective schools for decades, whereas the leadership at 
these schools and districts often changed rapidly. One teacher, in particular, stated that over a 5- to 6-year 
period, there was substantial turnover in leadership, particularly at the level of superintendents. Yet, the 
teacher had been at the school for 20 years, and had the same principal for about 15 years. In this case, the 
principal was an important part of the change process because he understood the culture of the school 
quite well. The teacher noted that, “he encouraged change, encouraged teaming, and actually forced a lot 
of change.” For example: 

Whenever anything new was introduced to our school, it was introduced in small groups 
to get a reaction. Then those small groups would take it to a larger group, then we would 
all come together... as the whole faculty and we would have a discussion about what we 
thought was right or wrong... but usually the way it was presented. There was really no 
objection to the whole program...! reckon it was the way it was presented to the whole 
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faculty. Most people would agree with the program, there might just be one or two little 
things that needed to be changed. 

The principal from this school was quite effective because he knew the system very well, and knew how 
to frame the problem to facilitate change. Another teacher from a different school comparing the 
leadership styles of two recent principals, shared: 

The previous principal pressured teachers to accomplish more and more. And as a result 
we won awards for being a Blue Ribbon School, a Super Safe School, and even the 
School of the State. The principal now does not ask anything of teachers and just wants 
everyone to be happy, satisfied, and feel like they are part of the decision making process. 

And I think there's a happy medium somewhere. 

According to many participants, the push for change comes from the principal. Administrative 
stability also seems to be associated with effectiveness in the minds of many participants. When asked 
what made their school so successful, the principal of one Blue Ribbon middle school replied: 

I believe that basically from about 1974 until now, there have only been three 
administrators in this building, so there’s been a lot of stability here. And what tends to 
happen in this building anyway is that we don’t give up on something until we find 
something better, like a continuous improvement, and that’s been since I was a teacher 
here in 1974. We’ve really tried to come up with something as a process and not touch it 
until we can improve the process. 
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Two main themes emerged from this segment of the interviews. First, although the staff members 
at the Blue Ribbon Schools were often innovative and talented, they tended to look toward the school 
leadership for deep-structural change efforts. Second, the leaders who were cited as most effective were 
those who were constantly looking to improve the process, but who were not interested in change for 
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change’s sake. They were reflective upon their actions and closely monitored the effectiveness of changes 
in practice. A key to this process is tied to the stability of the school leadership. 

Summary 

Overall, two major findings emerged from the structured interviews with principals and teachers 
from National Schools of Excellence. First, the data suggest that Anderson’s (1982) framework remains a 
useful way to think about school culture. The findings suggest that interpersonal interactions and attitudes 
and beliefs amongst the staff are the most important elements related to school effectiveness. Specifically, 
successful schools tend to be inhabited by constituents who are open to new ideas, willing to consider 
multiple points of view, willing to compromise, and flexible in their thinking — all attributes that could be 
associated with the creative subcomponent of the social structures dimension. Although these findings 
may seem intuitive, the results can be contrasted with the hard-nosed “my way or the highway” style of 
leadership that is often portrayed as affecting positive school change in films and other popular media (for 
reviews, see Joseph & Bumaford, 2001; Stemler, in press; Tan, 1999.). Furthermore, the results of the 
interviews suggest that, although the stability of the leadership may play an important part in determining 
school effectiveness, the key ingredient in terms of leadership is a push for change and improvement 
coupled with a commitment to following a course of action and adequate reflection upon the evidence. 

Teachers in schools who are faced with administrative changes every year or two are constantly 
subjected to new administrative visions and may not have adequate time to reflect upon the effectiveness 
of a particular course of action. One can easily imagine how the prospect of being faced with a new 
administrative mandate every few years could lead teachers to disengage from the decision-making 
process. Yet, such administrative changes are inevitable. The interview results indicate that the key 
element separating successful schools from the less successful ones is that the new administration is 
flexible, is willing to listen to various perspectives, and follows through on commitments. These 
characteristics create a more positive climate toward educational change. 
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Survey Results 



As of December 29, 2002, survey responses had been gathered from a total of 27 teachers and 3 
principals across six different schools. Due to technical difficulties associated with the primary online 
data collection, the data from six teachers and one principal from two unique schools were lost. However, 
the technical problems have been corrected and now the system is working properly. The data reported in 
this policy brief were drawn from 2 1 teachers associated with five schools, and the data-collection effort 
is still in progress. The teacher-level data were averaged across participants from each school so that one 
aggregate, school-level score on the SCI was computed for each school. Three of the schools participating 
in the project are elementary schools, one is a high school, and the remaining school is a middle/high 
school. The schools represent suburban and rural schools from the states of Pennsylvania, 2 Maine, and 
California. Table 4 presents the aggregate student achievement data for each of these schools. The test 
results used for each school correspond to the grade level mandated to participate in the state-level 
examination during the 2000-2001 session (typically grade 4 in the elementary school and grade 10 or 1 1 
at the high school). 



Table 4: Percentage of Students At or Above State Standards on Relevant State Tests in Reading and 
Mathematics 



School ID 


School Type 


Students At or Above 
Standard State Tests in 
Reading 


Students At or Above 
Standard State Tests in 
Mathematics 


i 


Rural ES 


63% 


16% 


2 


Suburban ES 


99% 


91% 


3 


Suburban ES 


49% 


55% 


4 


Rural MS/HS 


45% 


6% 


5 


Suburban HS 


63% 


41% 



Table 5 presents the zero-order correlation coefficients among the physical resource, social 
structure, and attitudes and beliefs subscale scores on the one hand, and the two outcome measures (i.e., 
the percentage of students at or above standards on the state test for reading and mathematics at each 
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school in our sample) on the other. The 0.10 alpha level was used as the criterion for establishing 
statistical significance at this point in the project. This alpha level was chosen because of the low 
statistical power arising from the small sample size and the desire to reduce the risk of Type II error. No 
statistically significant correlations appear between either of the achievement measures and the subscale 
scores. However, it should be kept in mind that, because the study is a work in progress, the sample size 
for the pilot study is very small (n=5 schools) and the analysis had extremely low statistical power. Yet, 
even with this small sample, the moderately high levels of the correlation coefficients are promising. 
Furthermore, a statistically significant relationship does appear between scores on the social structure 
dimension and the attitudes and beliefs dimension. Although the data are correlational and, therefore, 
causal inferences cannot be supported, the significant relationship is understandable considering that the 
nature of the schools’ staffs’ attitudes and beliefs probably has an impact upon the quality of the social 
interactions that transpire among various social groups. 



Table 5: Zero-Order Correlations Among Outcome Variables and Subscale Scores 





Reading 

Tests 


Math Tests 


Physical 

Resources: 

TOTAL 


Social 

Structures: 

TOTAL 


Attitudes and 
Beliefs: 
TOTAL 


Reading Tests 


i 










Math Tests 


0.78 


i 








Physical Resources: 
TOTAL 


0.44 


0.59 


i 






Social Structures: 
TOTAL 


0.54 


0.61 


0.04 


i 




Attitudes and Beliefs: 
TOTAL 


0.68 


0.48 


0.11 


0.90 


i 



Note: Bold values indicate that the correlation is significant at the 0.10 level (2 -tailed). 



2 The exam results for the State of Pennsylvania are reported by subdomain. For the present pilot analysis, the 
reading test scores used correspond to the Basic Understanding portion of the exam, and the mathematics test scores 
used correspond to the Problem Solving portion of the exam. 
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The Utility of the Theory of Successful Intelligence for Understanding School Effectiveness 

In order to test the utility of the successful intelligence theory for predicting schools’ 
effectiveness as intelligent systems, we next explored the correlations among each of the subcomponent 
scores and the outcome measures. Enhancing the findings presented in Table 5, the results presented in 
Table 6 indicate that several statistically significant relationships exist among the analytic, creative, and 
practical subcomponents of the social structure scale and the attitudes and beliefs scale. 

Table 6: Zero-Order Correlations Among Outcome Variables and Process Scores 





Reading 

Tests 


Math Tests 


SS: A 


SS: C 


SS: P 


AB: A 


AB: C 


AB: P 


PR: A 


PR: C 


PR: P 


Reading Tests 


1.00 






















Math Tests 


0.78 


1.00 




















SS: A 


0.32 


0.44 


1.00 


















SS: C 


0.92 


0.91 


0.45 


1.00 
















SS: P 


0.77 


0.71 


0.76 


0.88 


1.00 














AB: A 


0.92 


0.96 


0.40 


0.96 


0.76 


1.00 












AB: C 


0.80 


0.57 


0.76 


0.77 


0.94 


0.69 


1.00 










AB: P 


0.56 


0.37 


0.69 


0.65 


0.91 


0.45 


0.88 


1.00 








PR : A 


0.18 


0.15 


-0.71 


0.23 


-0.14 


0.17 


-0.31 


-0.17 


1.00 






PR: C 


0.17 


0.55 


-0.34 


0.40 


0.03 


0.41 


-0.26 


-0.19 


0.77 


1.00 




PR: P 


0.79 


0.60 


0.70 


0.83 


0.98 


0.70 


0.97 


0.94 


-0.13 


-0.10 


1.00 



Note: Bold values indicate that the correlation is significant at the 0.10 level (2-tailed). PR=Physical 
Resources, SS-Social Structures, AB=Attitudes and Beliefs, A=Analytic, C=Creative, and P=Practical 



Two findings from Table 6 are especially noteworthy. First, the scores from each of the practical 
subscales of each dimension are all statistically significantly correlated. This suggests that schools with a 
more practical orientation may tend to think practically across all of the domains that impact school 
effectiveness. 

Second, and perhaps more noteworthy, is the statistically significant relationship between the 
analytic subcomponent of the attitudes and beliefs scale and the percentage of students at or above 
standards in both mathematics and reading. In addition, a statistically significant relationship was also 
indicated between the creative subcomponent of the social structures scale and the percentage of students 
at or above standards in both mathematics and reading. Further, a statistically significant relationship was 
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detected between the practical subcomponent of the physical resources dimension and the percentage of 
students at or above standards in reading. When comparing these results to those presented in Table 5, the 
findings suggest that the theory of successful intelligence may provide a more sensitive measure of the 
key factors associated with school effectiveness than simply looking at the subscale scores in the 
aggregate. 

Although the results are quite preliminary and based upon a sample of five schools, the data 
suggest that using the theory of successful intelligence to understand the subcomponents of each key 
factor may yield greater information about the processes associated with greater school effectiveness. 
Specifically, the data suggest that the analytic subcomponent of the attitudes and beliefs dimension is 
strongly associated with school effectiveness. In practice, this finding might mean that more effective 
schools give their teachers more time for reflection or for observation of each other’s practice. 
Furthermore, the relationship between the creative subcomponent of the social structures scale and 
student achievement imply that more effective schools may be open to new ideas, and may have a culture 
where the discussion of new and innovative ideas is encouraged. Finally, the relationship between the 
practical subcomponent of the physical resources dimension and reading achievement implies that 
resources in effective schools may be used in ways that are more pragmatically oriented (a finding that 
was also noted during the structured interviews). 

The analytic component of the attitudes and beliefs scale is significantly correlated with the 
creative component of the social structure scale. One possible interpretation of this finding could be that 
both subcomponent scales are measuring the same construct. An alternative explanation could be that 
each subcomponent contributes to school effectiveness in a unique way. For example, an effective school 
might generate a host of new and innovative ideas, and then allow time for careful reflection upon those 
ideas. Although either of the above interpretations is plausible, the latter is consistent with Sternberg and 
Lubart’s (1995) investment theory of creativity. 

While no statistical control has been exerted over extraneous variables (such as student SES or 
professional qualifications of the surveyed teachers) in this relatively small-scale inquiry, the results of a 
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large-scale inquiry would be subjected to a regression analysis and the colinearity among these variables 
would be empirically estimated. Such an analytic procedure would not be appropriate for the pilot data 
given that the ratio of variables to sample size is quite high (a situation that would lead to model over- 
fitting and introduce the risk of severe shrinkage). 

The Utility of the Mineralogical Theory of School Modifiability for Understanding 

School Effectiveness 

The results presented in Table 7 correspond to the subscale scores associated with the various 
school profiles set forth by Sternberg’s mineralogical theory of school modifiability discussed earlier. 
Ideally, the profile scores are designed to act like a Thurstone scale, meaning that while certain elements 
of each profile would be present in all schools, each school would be most closely associated with only 
one dominant profile. Because of the way that schools were recruited for the pilot study, one would 
expect to find a disproportionate number of diamond in the rough school profiles. Schools that are not 
interested in actual change, or the appearance of change, would be predicted less likely to respond to our 
invitation to participate. 



Table 7: School Type, Student Achievement, and Mineralogical Profile Scores on the SCI. 



School 

ID 


School Type 


Students At or 
Above 
Standard for 
State Tests in 
Reading 


Students At 
or Above 
Standard for 
State Tests in 
Mathematics 


Rusted 

Iron 


Granite 


Amber 


Opal 


Cubic 

Zirconium 


Slightly 

Imperfect 

Diamond 


Lead 


Diamond 
in the 
Rough 


i 


Rural ES 


63% 


16% 


25.8 


32.5 


53.1 


60.0 


35.7 


46.0 


49.8 


73.3 


2 


Suburban ES 


99% 


91% 


27.1 


25.0 


50.0 


58.9 


46.4 


27.5 


50.0 


81.3 


3 


Suburban ES 


49% 


55% 


39.6 


41.7 


48.4 


69.6 


44.6 


47.5 


45.0 


62.5 


4 


Rural 

MS/HS 


45% 


6% 


58.4 


49.0 


53.2 


47.3 


43.4 


50.7 


58.2 


58.4 


5 


Suburban HS 


63% 


41% 


37.5 


25.0 


40.6 


46.4 


25.0 


35.0 


40.0 


83.3 



Note: Bold values indicate that the correlation is significant at the 0.10 level (2 -tailed). Scales range from 0-100. 



The existence of different profiles among five different schools indicates that the SCI was able to 
detect differences across the theoretical dimensions. Furthermore, the school profile scores presented in 
Table 7 seem to be operating in a way that is internally consistent with the mineralogical theory of school 
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modifiability. For example, the results in Table 7 indicate that the teachers in the high-achieving suburban 
elementary school in Pennsylvania identified most strongly with the variables associated with the 
diamond in the rough profile. Recall that the characteristics of diamond in the rough schools include a 
genuine commitment to change and educational experimentation, as well as positive staff attitudes and 
beliefs. The teachers from the suburban elementary school in California, however, identified most 
strongly with the variables associated with the opal profile. Recall that the characteristics of the opal 
school include an interest in the surface appearance of action and commitment to change, but lacking in a 
deep level of commitment to experimentation and change. Opal schools tend to be well resourced (a result 
that is corroborated by the school’s high score on the physical resources scale in Table 8), but they lack a 
certain level of commitment to educational improvement and experimentation (e.g., the school’s lower 
scores on the attitudes and beliefs scale in Table 8, and even lower scores on the creative component of 
this dimension in Table 9). 



Table 8: School Type, Student Achievement, and Subscale Scores on the SCI 3 



School ID 


School Type 


Students At or 
Above Standard 
State Tests in 
Reading 


Students At or 
Above Standard 
State Tests in 
Mathematics 


Overall Scale Scores 
(scale ranges from 0-100) 










Physical 

Resources 


Social 

Structures 


Attitudes 
and Beliefs 


1 


Rural ES 


63% 


16% 


76.8 


63.0 


68.7 


2 


Suburban ES 


99% 


91% 


79.0 


71.5 


74.5 


3 


Suburban ES 


49% 


55% 


80.5 


65.4 


60.8 


4 


Rural MS/HS 


45% 


6% 


53.9 


59.3 


56.6 


5 


Suburban HS 


63% 


41% 


59.0 


76.8 


78.9 



Note: Bold values indicate the highest subscale score for that school. 



3 The results presented in Table 8 represent the relative score on each dimension of the subscales. The scale ranges 
from 0-100 with higher numbers representing greater levels of physical resources, well-developed and active social 
structures, more positive attitudes and beliefs related to educational change and innovation, AND higher levels of 
general optimism among the staff. The values in bold represent the highest rated dimension for each particular 
school. The results have been rescaled in order to facilitate comparisons both across subscales and between schools. 
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Table 9: School Type, Student Achievement, and Process Scores for the Attitudes and Beliefs Factor of 
the SCI 



School ID 


School Type 


Students At or 
Above Standard 
State Tests in 
Reading 


Students At or 
Above Standard 
State Tests in 
Mathematics 


Overall Scale Scores for the 
Attitudes and Beliefs Dimension 
(scale ranges from 0-100) 










Analytic 


Creative 


Practical 


i 


Rural ES 


63% 


16% 


58.4 


68.8 


75.6 


2 


Suburban ES 


99% 


91% 


74.0 


81.3 


76.4 


3 


Suburban ES 


49% 


55% 


61.5 


60.9 


61.1 


4 


Rural MS/HS 


45% 


6% 


54.8 


60.9 


50.3 


5 


Suburban HS 


63% 


41% 


61.5 


81.3 


88.9 



Note: Bold values indicate the highest subscale score for that school. 



Table 7 also indicates that the rural middle/high school in Maine identified strongly with the 
variables associated with both the worst type of school (rusted iron) and the best type of school (diamond 
in the rough). However, the results may simply represent a moderate response set across all items, as the 
variability across mineralogical subtypes for this particular school is fairly restricted (ranging from 43.4 
to 58.4). Yet, this school had only 6% of ll th -grade students at or above standards on the state test in 
mathematics, and they also scored the lowest on all three subscales listed in Table 1. Furthermore, the 
teachers tended to endorse more of the characteristics of a rusted iron school than did any of the other 
schools in the pilot study. 

The results from this kind of analysis are helpful in diagnosing the extent to which a particular 
school may be open to new policy changes (i.e., modifiability). Recall that the opal school has a 
moderately poor prognosis for openness to surface-structural modifiability and a poor prognosis for deep- 
structural modifiability, whereas the diamond in the rough school has a good prognosis for both surface- 
structural and deep-structural modifiability. Furthermore, the mineralogical theory predicts, and the data 
from Table 8 support, the idea that where the opal school needs help is not in the area of physical 
resources, but rather in the area of staff attitudes and beliefs and social structural features. Consequently, 
policy decisions should be more targeted at these areas (e.g., fostering an environment that encourages 
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and rewards innovative ideas) in order to move the school from an opal to a diamond in the rough. By 
contrast, the rusted iron school has a poor prognosis for both surface-structural and deep-structural 
modifiability. Policy recommendations targeted at this type of school need to focus on all aspects of 
change (i.e., physical resource, social structures, and attitudes and beliefs). 

Table 10 presents the zero-order correlations among the various subscale scores related to each 
school profile. Although the statistical power is quite low, few statistically significant correlations were 
found among the different profiles. This situation is desirable from a theoretical standpoint, because each 
profile is supposed to be statistically independent of the others. In fact, theoretically, we would expect the 
strongest negative correlations to be associated with the rusted iron school, and the correlations to become 
more positive as one moves toward the diamond in the rough school. This trend begins to appear in our 
data — however, the pattern is broken with the slightly imperfect diamond school, whose scores are more 
significantly associated with the scores in the granite profile. This finding suggests that, if the school 
profiles represent an underlying continuum, then the slightly imperfect diamond school should be closer 
to the rusted iron school end of the continuum rather than the diamond in the rough end of the continuum. 



Table 10: Zero-Order Correlations Among Mineralogical Subscale Scores 





Reading 

Tests 


Math Tests 


SUB: 

RI 


SUB: 

GR 


SUB: 

AM 


SUB: 

OP 


SUB: 

CZ 


SUB: 

SID 


SUB: 

LD 


SUB: 

DIR 


Reading Tests 


i 




















Math Tests 


0.777 


i 


















SUB: R1 


-0.701 


-0.528 


1 
















SUB: GR 


-0.783 


-0.585 


0.796 


1 














SUB: AM 


-0.073 


-0.300 


0.103 


0.545 


1 












SUB: OP 


0.083 


0.412 


-0.454 


0.064 


0.277 


1 










SUB: CZ 


0.145 


0.318 


0.139 


0.463 


0.694 


0.528 


1 








SUB: SID 


- 0.908 


-0.800 


0.584 


0.880 


0.470 


0.111 


0.135 


1 






SUB: LD 


-0.124 


-0.378 


0.466 


0.629 


0.867 


-0.157 


0.624 


0.409 


1 




SUB: DIR 


0.755 


0.518 


-0.717 


- 0.990 


-0.577 


-0.202 


-0.530 


- 0.879 


-0.599 


1 



Note: Bold values indicate that the correlation is significant at the 0.10 level (2-tailed). SUB=Subscale Scores, 
RI=Rusted Iron, GR=Granite, AM=Amber, OP=Opal, CZ=Cubic Zirconium, SID=Slightly Imperfect Diamond, 
LD=Lead, DIR=Diamond in the Rough 





Summary 



The findings from the quantitative analyses support the utility of the successful intelligence 
theory for predicting the effectiveness of schools as intelligent systems. Specifically, these preliminary 
data suggest that the nine subcomponent scores may detect relationships that are masked by looking at the 
three overall subscale measures (i.e., physical resources, social structures, attitudes and beliefs). In 
addition, the data lend tentative support to the empirical validity of the mineralogical theory. This is an 
important step toward the construction of an empirical instrument capable of detecting which schools will 
be most likely to positively respond to surface-structural and deep-structural policy changes. 

POLICY IMPLICATIONS AND CONCLUSIONS 

The results of the data analysis presented in this policy brief are useful for addressing the three 
relevant policy issues of interest: (1) factors related to school effectiveness, (2) factors that are open to 
policy manipulation, and (3) characteristics of schools that are modifiable. First, the results of the 
structured interviews with principals and teachers from Blue Ribbon Schools suggest that the most basic 
element that comes together to make schools effective really has to do with the people and the nature of 
the social interactions. In general, the interview results show that people with strong interpersonal skills 
inhabit more effective schools. Specifically, both teachers and administrators at effective schools tend to 
be skilled at interacting not only with other teachers and administrators, but also with parents and students 
in a way that exhibits their open-mindedness, their flexibility, their willingness to listen to multiple points 
of view, and their willingness to compromise. However, most of these schools were already well 
resourced. Some teachers stated that, once a modicum of resource needs has been met, this situation freed 
up their time to think more creatively. A lack of adequate resources may negatively impact staff attitudes 
and beliefs and possibly affect the quality of social interactions in the school. 

Second, the subcomponent scores based upon the successful intelligence theory help to highlight 
the key areas of a particular dimension that are associated with student achievement and potentially open 
to policy manipulation. For example, the preliminary results here suggest that more effective schools are 
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characterized by higher analytic attitudes and beliefs, which may take the form of giving teachers time to 
reflect upon their practice and observe each others’ classroom practices. Similarly, the creative aspect of 
the social structures dimension was significantly associated with student achievement, possibly 
suggesting that more effective schools tend to create a safe environment for discussing innovative ideas. 
The interview results presented here suggest that, as a policy matter, schools are well-advised to bring on 
board teachers who are committed to the shared philosophy of the school. Furthermore, the results 
support the utility of the theory of successful intelligence for predicting the effectiveness of schools as 
intelligent systems. By looking at the subcomponents of each major dimension of school effectiveness, it 
is possible to detect relationships that would be missed if the information were examined only in the 
aggregate. 

Third, based on the data and impressions from the interviews with teachers and principals from 
successful schools, two common elements are related to their openness to change. First, although the 
school may be populated by energetic personnel, deep-structural changes are initiated by the school 
leadership. Furthermore, successful schools are typically characterized by a principal or superintendent 
who has been with the system for a fair amount of time (greater than 5 years), who tends to push for 
change, who remains committed to a particular course of action, and who reflects upon the evidence 
related to the effectiveness of his or her current practices. In addition, the data suggest that the school 
profiles set forth by the mineralogical theory are largely statistically independent. What is more, at least 
some school profiles appear to be associated with school effectiveness in the way that is consistent with 
the theory. This result is quite promising in light of the fact that the field of school-effectiveness research 
has been without a clear theoretical framework since its inception. Furthermore, the results from this kind 
of analysis may be helpful in diagnosing the extent to which different kinds of policy recommendations 
(e.g., those targeted at surface-structural changes versus those targeted at deep-structural changes) are 
likely to fall within a particular profile of school. 

The data in this pilot study are suggestive. A large-scale replication (an activity that is scheduled 
to take place during the next 3 years) is required before support policy recommendations can be made 
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from the data gathered by this project. Nevertheless, the results are encouraging, and it is clear that there 
are a number of important policy issues that may be addressed through this line of inquiry. 
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